Testing the convexity of the support of a distribution

Given n independent, identically distributed random vectors in Rd, drawn from a common density f , one wishes to find out whether the support of f is convex or not. In this paper we describe a test which decides correctly for sufficiently large n, with probability 1, whenever f is bounded away from zero in its compact support. We also show that the assumption of boundedness is necessary. The proposed test statistic is a second-order U -statistic with a random kernel. Moreover, we suggest a way of approximating the distribution of the test statistic under the null hypothesis (i.e., under convexity of the support). The performance of the proposed method is illustrated on simulated data sets.